feat(openai): Instrument structured outputs (chat.completions.parse) by Nik-Reddy · Pull Request #4416 · open-telemetry/opentelemetry-python-contrib

Nik-Reddy · 2026-04-13T00:17:45Z

Description

The OpenAI v2 instrumentation currently wraps chat.completions.create() but not chat.completions.parse(). The parse() method is used for structured outputs. Calls to parse() generate zero telemetry even when instrumentors are configured.

This PR adds instrumentation for both sync and async parse() methods, reusing the existing chat completion wrapper logic.

Fixes #3449

Changes

Added _is_parse_supported() version guard and wrap/unwrap calls for parse methods
Handle response_format being a Python type by recording json_schema as the output type attribute
8 new test cases (sync + async x content capture x semconv variants)
4 VCR cassettes for structured output calls

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

All 8 new tests pass. All 84 existing tests continue to pass.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added

MikeGoldsmith

Looking good, thanks @Nik-Reddy. I've left some suggestions and please can you add a changelog entry.

Nik-Reddy · 2026-04-14T21:45:22Z

Hi @MikeGoldsmith, I've addressed all three of your review comments:

Cached the result of _is_parse_supported() as self._parse_supported during _instrument()
Extracted shared test definitions into structured_outputs_utils.py
Added clarifying comment about parse/create relationship

Would appreciate a re-review when you get a chance. Thanks!

Nik-Reddy · 2026-04-15T01:09:15Z

Rebased on latest main. All review feedback @MikeGoldsmith addressed:

Cached _is_parse_supported() result on the instrumentor instance
Extracted shared test definitions into structured_outputs_utils.py
Added clarifying comment for parse vs create dispatch

MikeGoldsmith · 2026-04-15T18:55:12Z

Thanks for the updates, the changes look good. One thing still to address — there are unused imports in test_structured_outputs.py (import json and server_attributes as ServerAttributes) as flagged by @dehanjl. You can clean these up by running the following locally:

uv run tox -e lint

This needs to be done before we can accept.

Also, I want to raise something directly: the responses to review feedback have been very fast and follows a pattern that suggests an automated agent may be posting comments on your behalf. As per our AGENTS.md at the root of this project, discussions on OpenTelemetry repositories are for humans only — AI-generated comments on issues and PRs are not permitted. Please ensure that all review responses and PR comments are written and posted by you directly. Thanks for understanding.

Nik-Reddy · 2026-04-16T07:40:30Z

@dehanjl good catch on the unused imports. Cleaned those up, dropped json, ServerAttributes, EXPECTED_RESPONSE_CONTENT, and pytest (wasn't actually needed in the sync test file). Ran ruff locally and everything's passing now.

Rebased on latest main as well.

MikeGoldsmith

Ci is failing because of a new test that was added. precommit check is still failing too.

Please take a look 👍🏻

Nik-Reddy · 2026-04-29T04:34:56Z

@MikeGoldsmith, addressed all review feedback, including cached parse support check, shared test utilities, and semconv constants as per @lmolkova’s suggestion, along with the CI fixes. I will rebase on main today. Please let me know if any further adjustments are needed

Copilot

Pull request overview

Adds OpenAI v2 structured-output (chat.completions.parse) coverage to the existing OpenAI instrumentation so parse() calls emit the same telemetry as create().

Changes:

Wrap Completions.parse / AsyncCompletions.parse behind a version/feature guard in the instrumentor.
Extend request-attribute extraction to handle response_format passed as a Python type (e.g., a Pydantic model class).
Add sync/async structured output tests and VCR cassettes; update changelog entry.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
instrumentation-genai/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/init.py	Adds `parse()` support detection and instruments/uninstruments sync + async `parse()` methods.
instrumentation-genai/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/utils.py	Updates request attribute handling for `response_format` when it is a Python type.
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_structured_outputs.py	Adds sync `parse()` structured outputs tests (content/no-content; semconv variants).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_async_structured_outputs.py	Adds async `parse()` structured outputs tests (content/no-content; semconv variants).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/structured_outputs_utils.py	Adds shared prompt + Pydantic model for structured output tests.
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_structured_output_with_content.yaml	Recorded structured output cassette (sync, content).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_structured_output_no_content.yaml	Recorded structured output cassette (sync, no content).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_async_structured_output_with_content.yaml	Recorded structured output cassette (async, content).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_async_structured_output_no_content.yaml	Recorded structured output cassette (async, no content).
CHANGELOG.md	Documents the added `parse()` instrumentation feature.

Comments suppressed due to low confidence (1)

instrumentation-genai/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/utils.py:377

In create_chat_invocation(), when response_format is a plain string you set GenAIAttributes.GEN_AI_OPENAI_REQUEST_RESPONSE_FORMAT even on the new (util-genai / latest-experimental) path, while get_llm_request_attributes() uses gen_ai.output.type for latest-experimental. This makes the emitted attributes inconsistent and can drop gen_ai.output.type entirely for callers that pass response_format as a string. Consider setting GenAIAttributes.GEN_AI_OUTPUT_TYPE here as well (and mapping known values to the semconv enum where applicable).

    if (response_format := get_value(kwargs.get("response_format"))) is not None:
        # response_format may be string, object with a string in the `type` key,
        # or a type (e.g. Pydantic model class used with parse())
        if isinstance(response_format, type):
            invocation.attributes[GenAIAttributes.GEN_AI_OUTPUT_TYPE] = (
                GenAIAttributes.GenAiOutputTypeValues.JSON.value
            )
        elif isinstance(response_format, Mapping):
            if (
                response_format_type := get_value(response_format.get("type"))
            ) is not None:
                invocation.attributes[GenAIAttributes.GEN_AI_OUTPUT_TYPE] = (
                    response_format_type
                )
        else:
            invocation.attributes[
                GenAIAttributes.GEN_AI_OPENAI_REQUEST_RESPONSE_FORMAT
            ] = response_format

lzchen · 2026-05-02T19:16:55Z

@Nik-Reddy

Might have to rebase again.

Nik-Reddy · 2026-05-02T22:47:33Z

Pull request overview

Adds OpenAI v2 structured-output (chat.completions.parse) coverage to the existing OpenAI instrumentation so parse() calls emit the same telemetry as create().

Changes:

Wrap Completions.parse / AsyncCompletions.parse behind a version/feature guard in the instrumentor.

Extend request-attribute extraction to handle response_format passed as a Python type (e.g., a Pydantic model class).

Add sync/async structured output tests and VCR cassettes; update changelog entry.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
instrumentation-genai/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/init.py Adds parse() support detection and instruments/uninstruments sync + async parse() methods.
instrumentation-genai/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/utils.py Updates request attribute handling for response_format when it is a Python type.
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_structured_outputs.py Adds sync parse() structured outputs tests (content/no-content; semconv variants).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_async_structured_outputs.py Adds async parse() structured outputs tests (content/no-content; semconv variants).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/structured_outputs_utils.py Adds shared prompt + Pydantic model for structured output tests.
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_structured_output_with_content.yaml Recorded structured output cassette (sync, content).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_structured_output_no_content.yaml Recorded structured output cassette (sync, no content).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_async_structured_output_with_content.yaml Recorded structured output cassette (async, content).
instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_async_structured_output_no_content.yaml Recorded structured output cassette (async, no content).
CHANGELOG.md Documents the added parse() instrumentation feature.
Comments suppressed due to low confidence (1)

@lmolkova, addressed the 3 Copilot suggestions, participants is now list[str], and I added assert response.choices[0].message.parsed is not None in both sync and async tests so we catch it if the wrapper ever breaks the parsed return.

Besides, wanted to check, is that level of assertion enough, or would you prefer something that also validates the shape of the parsed object?

linux-foundation-easycla · 2026-05-05T04:18:28Z

The committers listed above are authorized under a signed CLA.

✅ login: Nik-Reddy / name: Nik-Reddy (0374f33, 0c5451a, 20505f6, 4286131, 60c66e3, c180326, ebbdc72, fb4a084)

Fixes open-telemetry#3449

Nik-Reddy · 2026-05-05T05:10:07Z

@Nik-Reddy

Might have to rebase again.

@lzchen rebased on latest main. Also fixed the parse() wrappers after the positional args change in #4445 and the content_mode removal in #4315. CI should be clean now.

…rebase

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Nik-Reddy · 2026-05-06T23:12:55Z

@lzchen Rebased and addressed all copilot review comments, looks like the full test suite hasn't kicked off yet though.

Nik-Reddy requested a review from a team as a code owner April 13, 2026 00:17

github-project-automation Bot added this to Python PR digest Apr 13, 2026

github-actions Bot assigned lmolkova Apr 13, 2026

github-actions Bot requested a review from lmolkova April 13, 2026 00:17

Nik-Reddy mentioned this pull request Apr 13, 2026

Instrument OpenAI structured outputs #3449

Open

xrmx added the gen-ai Related to generative AI label Apr 13, 2026

MikeGoldsmith requested changes Apr 13, 2026

View reviewed changes

github-project-automation Bot moved this to Reviewed PRs that need fixes in Python PR digest Apr 13, 2026

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from 3ec82e7 to 6d00a12 Compare April 13, 2026 20:16

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from 6d00a12 to ce2065f Compare April 15, 2026 01:08

Nik-Reddy requested a review from MikeGoldsmith April 15, 2026 01:16

dehanjl reviewed Apr 15, 2026

View reviewed changes

Comment thread instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_structured_outputs.py

Nik-Reddy requested a review from dehanjl April 16, 2026 03:45

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from 3cbb955 to 9a7ae07 Compare April 16, 2026 07:37

MikeGoldsmith requested changes Apr 16, 2026

View reviewed changes

Comment thread instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_structured_outputs.py

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch 2 times, most recently from 208bc51 to 5db9b7a Compare April 16, 2026 21:00

lmolkova reviewed Apr 16, 2026

View reviewed changes

Comment thread ...opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/utils.py Outdated

Nik-Reddy requested review from MikeGoldsmith and lmolkova April 24, 2026 20:39

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from 1ad49a1 to 6867155 Compare April 29, 2026 17:45

lmolkova requested review from Copilot and removed request for dehanjl May 1, 2026 01:40

Copilot started reviewing on behalf of lmolkova May 1, 2026 01:41 View session

Copilot AI reviewed May 1, 2026

View reviewed changes

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch 3 times, most recently from 5887c9e to 4bec562 Compare May 2, 2026 22:37

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from 71ad47e to 8d79b51 Compare May 5, 2026 04:18

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from 8d79b51 to bad8407 Compare May 5, 2026 04:18

Nik-Reddy added 6 commits May 4, 2026 22:03

feat(openai): Instrument structured outputs (chat.completions.parse)

4286131

Fixes open-telemetry#3449

docs: Add CHANGELOG entry for structured outputs instrumentation

0c5451a

fix: remove unused imports in structured outputs tests

0374f33

fix: formatting and pylint issues in structured outputs

20505f6

fix output type semconv value and run ruff format

c180326

use semconv constants for output type values

fb4a084

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from bad8407 to 76ee223 Compare May 5, 2026 05:03

align parse() wrapping with positional args after open-telemetry#4445 …

ebbdc72

…rebase

Nik-Reddy force-pushed the feat/openai-structured-outputs-3449 branch from aaf42fb to ebbdc72 Compare May 6, 2026 05:33

Nik-Reddy requested a review from Copilot May 6, 2026 05:35

Copilot started reviewing on behalf of Nik-Reddy May 6, 2026 05:36 View session

Copilot AI reviewed May 6, 2026

View reviewed changes

Comment thread instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_structured_outputs.py

Comment thread ...ntation-genai/opentelemetry-instrumentation-openai-v2/tests/test_async_structured_outputs.py

skip structured output tests on openai < 1.40.0

60c66e3

Conversation

Nik-Reddy commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Type of change

How Has This Been Tested?

Checklist:

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nik-Reddy commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nik-Reddy commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MikeGoldsmith commented Apr 15, 2026

Uh oh!

Nik-Reddy commented Apr 16, 2026

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Nik-Reddy commented Apr 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lzchen commented May 2, 2026

Uh oh!

Nik-Reddy commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request overview

Reviewed changes

Uh oh!

linux-foundation-easycla Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nik-Reddy commented May 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Nik-Reddy commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Nik-Reddy commented Apr 13, 2026 •

edited

Loading

Nik-Reddy commented Apr 14, 2026 •

edited

Loading

Nik-Reddy commented Apr 15, 2026 •

edited

Loading

Nik-Reddy commented May 2, 2026 •

edited

Loading

linux-foundation-easycla Bot commented May 5, 2026 •

edited

Loading